Two-Sided Exponential Concentration Bounds for Bayes Error Rate and Shannon Entropy
نویسندگان
چکیده
We provide a method that approximates the Bayes error rate and the Shannon entropy with high probability. The Bayes error rate approximation makes possible to build a classifier that polynomially approaches Bayes error rate. The Shannon entropy approximation provides provable performance guarantees for learning trees and Bayesian networks from continuous variables. Our results rely on some reasonable regularity conditions of the unknown probability distributions, and apply to bounded as well as unbounded variables.
منابع مشابه
Weak Gibbs Measures: Speed of Convergence to Entropy, Topological and Geometrical Aspects
Abstract. In this paper we obtain exponential large deviation bounds in the Shannon-McMillan-Breiman convergence formula for entropy in the case of weak Gibbs measures and topologically mixing subshifts of finite type. We also prove almost sure estimates for the error term in the convergence to entropy given by Shannon-McMillan-Breiman formula for both uniformly and non-uniformly expanding shif...
متن کاملEntropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy
We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound’s prior, violating the hypothesis of the PAC-Bayes theorem that the prior is cho...
متن کاملComparison of Estimates Using Record Statistics from Lomax Model: Bayesian and Non Bayesian Approaches
This paper address the problem of Bayesian estimation of the parameters, reliability and hazard function in the context of record statistics values from the two-parameter Lomax distribution. The ML and the Bayes estimates based on records are derived for the two unknown parameters and the survival time parameters, reliability and hazard functions. The Bayes estimates are obtained based on conju...
متن کاملEstimation and Hypothesis Testing for Exponential Lifetime Models with Double Censoring and Prior Information
In this paper, on the basis of a doubly censored sample and in a Bayesian framework, the problem of estimating the mean lifetime, hazard rate, and survival function of the exponential lifetime model is addressed. Bayes estimators under squared-error loss functions are obtained in closed forms. Highest posterior density (HPD) estimators and credible intervals are computed using iterative methods...
متن کاملRedundancy-Related Bounds on Generalized Huffman Codes
This paper presents new lower and upper bounds for the compression rate of optimal binary prefix codes on memoryless sources according to various nonlinear codeword length objectives. Like the most well-known redundancy bounds for minimum (arithmetic) average redundancy coding — Huffman coding — these are in terms of a form of entropy and/or the probability of the most probable input symbol. Th...
متن کامل